Search for: All records

There is a growing need to empirically evaluate the quality of online instructional interventions at scale. In response, some online learning platforms have begun to implement rapid A/B testing of instructional interventions. In these scenarios, students participate in series of randomized ex- periments that evaluate problem-level interventions in quick succession, which makes it difficult to discern the effect of any particular intervention on their learning. Therefore, dis- tal measures of learning such as posttests may not provide a clear understanding of which interventions are effective, which can lead to slow adoption of new instructional meth- ods. To help discern the effectiveness of instructional in- terventions, this work uses data from 26,060 clickstream se- quences of students across 31 different online educational experiments exploring 51 different research questions and the students’ posttest scores to create and analyze different proximal surrogate measures of learning that can be used at the problem level. Through feature engineering and deep learning approaches, next problem correctness was deter- mined to be the best surrogate measure. As more data from online educational experiments are collected, model based surrogate measures can be improved, but for now, next prob- lem correctness is an empirically effective proximal surrogate measure of learning for analyzing rapid problem-level exper- iments.